PARTIAL FULFILLMENT OF THE REQUIREMENTS for the degree DOCTOR OF PHILOSOPHY in STATISTICS

Recent work in the areas of nonparametric regression and spatial smoothing has focused on modelling functions of inhomogeneous smoothness. In the regression literature, important progress has been made in fitting free-knot spline models in a Bayesian context, with knots automatically being placed more densely in regions of the covariate space in which the function varies more quickly. In the spatial statistics literature, attention has focused on using nonstationary covariance structures to account for inhomogeneity of the spatial field. In this dissertation, I use nonstationary covariance functions in Gaussian process (GP) prior distributions over functions to perform both nonparametric regression and spatial smoothing in a Bayesian fashion. I extend the kernel convolution method of Higdon et al. (1999) to create a class of nonstationary covariance functions. I prove that the nonstationary covariance functions retain the differentiability properties of the stationary correlation functions on which they are based, provided there is sufficient smoothness in the underlying kernel structure used to generate the nonstationarity. The stationary Matérn covariance function has desirable differentiability properties; the generalized kernel convolution method developed here provides a Matérn-based nonstationary covariance function. I develop a generalized nonparametric regression model and assess difficulties in identifiability and in fitting of the model using Markov Chain Monte Carlo (MCMC) algorithms. Of particular note, I show how to improve MCMC performance for non-Gaussian data based on an approximate conditional posterior mean. The modelling approach produces a flexible response surface that responds to inhomogeneity while naturally controlling overfitting. For Gaussian errors, on test datasets in one dimension, the GP model performs well, but not as well as the free-knot spline method. However, in two and three dimensions, the nonstationary GP model seems to outperform